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Description 

FIELD OF THE INVENTION 

[0001] The present invention relates to computer in- 
put devices generally and more particularly to handwrit- 
ing recognition computer input devices. 

BACKGROUND OF THE INVENTION 

[0002] There exists a significant amount of activity in 
the field of on-line handwriting recognition. The prior art 
current to 1 990 is reviewed in "The State of the Art in 
On-Line Handwriting Recognition" by Charles C. Tap- 
pert et at, IEEE Transactions on Pattern Analysis and 
Machine Intelligence, Vol. 12, No. 8, August, 1990. 
[0003] Generally speaking, on-line handwriting anal- 
ysis is currently employed for two distinct applications: 
identity verification and input of handwritten letters and 
numbers into a computer. These two applications have 
sharply contrasting operational requirements and goals. 
Handwriting analysis for identity verification senses fea- 
tures of handwriting which are distinct for each individual 
and thus can be used to unambiguously identify a given 
individual. In contrast, handwriting analysis for alphanu- 
meric input to a computer seeks to minimize the effect 
of the very features which are important for identity ver- 
ification and to concentrate on universal handwriting 
characteristics which can be associated with given sym- 
bols independently of the individual writer. 
[0004] Currently existing and proposed systems pro- 
viding handwriting analysis for alphanumeric input to a 
computer are generally geared towards recognition of 
how a symbol looks rather than how it is created. Ac- 
cordingly, such systems employ digitizers or graphic 
tablets. 

[0005] Signature verification systems, on the other 
hand, attempt to identify biometric characteristics of the 
writer and employ indications such as pressure and ac- 
celeration during writing. 

[0006] U.S. Patent 4,345,239 employs pen accelera- 
tion for use in a signature verification system. U.S. Pat- 
ent 5,054,088 employs both acceleration and pressure 
data characteristics of handwriting for identity verifica- 
tion. As indicated by the above patents, pen accelera- 
tion is employed for signature verification because it is 
a personal feature, characteristic of each individual. Ac- 
cordingly, pen acceleration has not been employed for 
alphanumeric input. 

[0007] U.S. Patent 4,751,741 describes pen-type 
character recognition apparatus which employs pen 
pressure data exclusively. 

[0008] U.S. Patent 4,695,741 describes a light-pen 
equipped with an accelerometer for detecting tapping of 
the pen on a computer screen. As indicated by the 
above patent, pen acceleration is not used more gener- 
ally to determine contact between a pen and a surface. 
[0009] U.S. Patent 4,845,684 describes a hand-held 



ultrasonic apparatus for detecting contact between a 
pen and a writing surface. The above patent does not 
disclose use of acceleration data. 
[0010] US-A-41 22435 describes an apparatus using 
5 accelerometers whic, when employed to write on a spe- 
cial surface, produces signals which can be used for 
handwriting analysis. 

[001 1 ] Position digitization through ultrasonic position 
digitisation is well known in the prior art, as for example 

10 in U.S. Patents 3,731,273; 4,317,672; 4,578,674; 
4,654,648; 4,758,691; 4,814,552; 4,862,152; 
4,991,148; 5,142,506; 5,214,615; 5,239,139; and 
5,280,457. Other publications describing ultrasonic po- 
sition digitisation include "A sonic pen: a digital stylus 

15 system" by Brenner and DeBruyne, IEEE Transactions 
on Computers, vol. C-19, No. 6, June 1970, pp. 
546-548; "Compact Large-Area Graphic Digitizer for 
PC" by DeBruyne, IEEE Computer Graphics and Appli- 
cation, December 1986, pp. 49-53; IBM Technical Dis- 

20 closure Bulletin, vol. 1 2, No. 3, August 1 969, p. 390; and 
"US cursor position detection", IBM Technical Disclo- 
sure Bulletin, vol. 27, No. 11, April 1985. 
[0012] In practice, digitized positions derived from ul- 
trasonic measurement are not determined precisely 

25 enough to allow even minimally operative handwriting 
recognition, so that handwriting recognition from ultra- 
sonic position detection is not known in the prior art. 
[0013] SU-A-1 357986 discloses a graphical data 
read-out unit having acceleration sensors located in one 

30 plane symmetrical to a casing axis. A writing element 
has a pressure sensor in the tip thereof and perpendic- 
ular to the casing axis. 

[001 4] In the Journal of the Institute of Electronics, In- 
formation and Communication Engineers D-l (Japan) 

35 1 993, there is disclosed a method of extraction of char- 
acter information from a pen which uses two accelera- 
tion sensors to measure x and y component of the pen 
movement. A pen-touching signal is generated by a 
small high frequency acceleration signal caused by fric- 

^o tion between the pen and the paper. 

[0015] The present invention seeks to provide an im- 
proved computer input device. 

[001 6] Accordingly, the invention provides a handwrit- 
ing recognition apparatus comprising handwriting input 
45 apparatus employs at least two different sensing tech- 
niques to sense handwriting and including at least one 
accelerometer located in. said hand held writing imple- 
ment other than at a tip thereof; and symbol identifica- 
tion apparatus receiving an output of the handwriting in- 
50 put apparatus for providing an output indication of sym- 
bols represented by the handwriting, and wherein said 
symbol identification apparatus includes a first identifi- 
cation channel for employing an input from said hand- 
writing input apparatus to provide a first output repre- 
ss senting identification of symbols, a second identification 
channel for employing an input from said at least one 
accelerometer to provide a second output representing 
identification of symbols, and symbol selection appara- 
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tus for selecting between the first and the second out- 
puts based on predetermined criteria. 
[0017] The handwriting input apparatus may compris- 
es a position digitizer. 

[001 8] The at least one accelerometer may comprise 5 
a plurality of mutually orthoganally disposed accelerom- 
eters. 

[0019] The symbol identification apparatus may in- 
clude combined position and acceleration processing 
apparatus, which combine the inputs from the digitizer *o 
and said at least one accelerator in identifying symbols. 
[0020] Handwriting recognition apparatus as claimed 
in claim 1, wherein said handwriting input apparatus 
comprises ultrasonic position determination apparatus 
operative in air. J 5 

[0021] The handwriting input apparatus may be oper- 
ative to write on a computer screen. 
[0022] The invention also provides a handwriting rec- 
ognition method comprising employing at least one ac- 
celerometer located in a hand writing implement other 20 
than at the tip thereof and at least one additional differ- 
ent sensing technique to sense handwriting; and receiv- 
ing an output indication of the handwriting and providing 
an output indication of symbols represented by the 
handwriting, wherein said step of providing an output in- 25 
dication of symbols employs input from a digitizer to pro- 
vide a first output representing identification of symbols 
along a first identification channel and employs input 
from said at least one accelerometer to provide a sec- 
ond output representing identification of symbols along 30 
a second identification channel and selects between the 
first and second outputs based on predetermined crite- 
ria. 

[0023] The at least one additional sensing technique 
may comprises position digitizing. 35 
[0024] The method may further comprise identifying 
symbols by combining digitized position inputs and 
sensed acceleration information. 
[0025] The step of receiving may include, employing 
digitized position inputs to provide a first output repre- *o 
senting identification symbols, employing acceleration 
information to provide a second output representing 
identification of symbols, and selecting between said 
first and second outputs. 

45 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0026] The present invention will be understood and 
appreciated from the following detailed description, tak- 
en in conjunction with the drawings in which: so 

Fig. 1 A is a simplified semi-pictorial semi-block di- 
agram illustration of handwriting recognition appa- 
ratus constructed and operative in accordance with 
a preferred embodiment of the present invention; ss 
Fig. 1 B is a simplified semi-pictorial semi-block di- 
agram illustration of handwriting recognition appa- 
ratus constructed and operative in accordance with 



an alternative preferred embodiment of the present 
invention; 

Figs. 2A through 2H are schematic drawings of pre- 
ferred structures of portions of the apparatus of 
Figs. 1A and 1B; 

Fig. 3 is a simplified block diagram of the handwrit- 
ing recognition apparatus of Figs. 1A and 1B; 
Fig. 4A is a simplified flow chart illustrating the 
teaching process performed by the handwriting rec- 
ognition apparatus of Figs. 1A and 1B; 
Fig. 4B is a simplified flow chart illustrating the rec- 
ognition process performed by the handwriting rec- 
ognition apparatus of Figs. 1A and 1B; 
Figs. 5A through 5 D are graphical depictions of data 
illustrating a preferred method for a portion of the 
teaching and recognition processes performed by 
the handwriting recognition apparatus of Figs. 1A 
and 1B; 

Fig. 6A is a simplified illustration of a preferred me- 
chanical structure of the digitizing pen/accelerome- 
ter of Fig. 1 A; 

Fig. 6B is a simplified illustration of a preferred me- 
chanical structure of the digitizing pen/accelerome- 
ter of Fig. 1 B; 

Fig. 7 is a simplified semi-pictorial semi-block dia-' 
gram of handwriting recognition apparatus con- 
structed and operative in accordance with another 
alternative preferred embodiment of the present in- 
vention; 

Fig. 8 is a simplified block diagram of the handwrit- 
ing recognition apparatus of Fig. 7; * 
Fig. 9A is a simplified flow chart illustrating the 
teaching process performed by the handwriting rec- > 
ognition apparatus of Fig. 7; and 
Fig. 9B is a simplified flow chart illustrating the rec- 
ognition process performed by the handwriting rec- 
ognition apparatus of Fig. 7. 

DETAILED DESCRIPTION OF PREFERRED 
EMBODIMENTS 

[0027] Reference is now made to Fig. 1 A which is a 
simplified pictorial illustration of a handwriting recogni- 
tion apparatus constructed and operated in accordance 
with the present invention. A digitizing. pen/accelerom- 
eter 10 is employed to write on a graphic tablet 12. 
[0028] Pen 1 0 also comprises a built-in accelerometer 
25, such as model ICS 3031 -2 commercially available 
from IC Sensors, 1701 McCarthy Blvd., Milpitas, CA 
95035. Alternatively, pen 10 may comprise a plurality of 
accelerometers, which accelerometers may or may not 
be orthogonally disposed. 

[0029] Pen 1 0 also comprises one or more amplifiers 
30, associated with the one or more accelerometers 25. 
Fig. 2A is a schematic drawing of a preferred embodi- 
ment of amplifier 30 suitable for use in this application. 
[0030] Alternatively, pen 1 0 may be fitted with retrofit- 
table apparatus comprising one or more accelerometers 
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25 as described above, and also comprising associated 
amplifiers 30, as described above. The apparatus may 
have the form of a cap fitting the end of pen 10, a ring 
encircling the pen, or any other suitable form. 
[0031 ] Graphic tablet 1 2 senses the position of the tip 
of pen 10 relative to tablet 12 and provides a digitized 
output of data describing the position. Graphic tablet 1 2 
may be any suitable graphic tablet, such as a Wacom 
Tablet Processor model SD-510C available from WA- 
COM Computer Systems GmbH, Hellersbergstrasse 4, 
W-4040 Neuss 1 , Germany. 

[0032] The data from the one or more accelerometers 
in pen 10 is termed herein "accelerometer data". Accel- 
erometer data and location data from graphic tablet 12 
are each sent through separate cables to a control cir- 
cuit 40. Alternatively, the accelerometer data and loca- 
tion data may be sent through a single cable. According 
to a still further alternative, the accelerometer data and 
location data may be sent through any kind of appropri- 
ate communication link, such as ultrasonic, infrared, or 
by any other suitable means. 

[0033] Control circuit 40 amplifies the acceleration 
signals from pen 1 0 and converts them to digital form, 
preferably using an analog to digital converter. Fig. 2B 
is a schematic drawing of a preferred embodiment of an 
analog to digital converter suitable for use in this appli- 
cation. 

[0034] Control circuit 40 then sends combined syn- 
chronized acceleration data and position data to an ac- 
celeration and position data processor 50. Data proces- 
sor 50 may be any suitable computer, for example, an 
IBM PC compatible computer with an 80386 processor 
chip. 

[0035] Associated with data processor 50 are a sym- 
bol display 60 and an input device 70. Recognition oc- 
curs within data processor 50 and the recognized sym- 
bols are displayed on symbol display 60. If necessary, 
the user may enter corrections of recognition errors us- 
ing input device 70. Alternatively, the user may enter 
corrections using pen 10 and digitizing tablet 12 to 
choose among alternatives displayed on the symbol dis- 
play, or using other suitable means. 
[0036] The functionality of the apparatus of Fig. 1A 
will now be described. The user employs pen 1 0 to write 
symbols on graphic tablet 12. Pen 10 sends accelera- 
tion data describing the accelerations of pen 10 during 
writing to control circuit 40. Graphic tablet 1 2 sends pen 
point position data describing the position over time of 
the point of pen 10 to control circuit 40. 
[0037] Control circuit 40 amplifies and digitizes the ac- 
celeration data. If the acceleration data and the position 
data are not synchronized in time, control circuit 40 syn- 
chronizes the acceleration and position data by match- 
ing pen status signals with acceleration data. Pen status 
signals are described more fully below with reference to 
Fig. 4B. The combined synchronized data is sent by 
control circuit 40 to data processor 50. 
[0038] Alternatively, control circuit 40 may also com- 



prise a processor and memory suitable for storing the 
combined synchronized data. In this case, pen 10, 
graphic tablet 12, and control circuit 40 may be used to 
produce and store the combined synchronized data. 
5 Then, at a later time, the data is sent by control circuit 
40 to data processor 50 in order for handwriting recog- 
nition to occur. 

[0039] Before handwriting recognition may occur, da- 
ta processor 50 must create a database which compris- 
10 es data about each symbol which may be recognized, 
specific to each person who uses the apparatus. The 
database is termed herein the per-person per-symbol 
database. 

[0040] During teaching, data processor 50 controls 
15 the creation of the per-person per-symbol database of 
handwriting characteristics. The per-person per-symbol 
database comprises data for each person whose hand- 
writing is known to the system, describing one or more 
prototypes for each symbol. Each prototype comprises 
20 parameters which describes the symbol as written by 
the person; the parameters are described in more detail 
below with reference to step 240. 
[0041] During teaching the user writes symbols from 
a pre-arranged script known to data processor 50, and 
25 the associated symbols appear on symbol display 60 
during the writing. Preferably, the pre-arranged script 
contains several repetitions of each symbol. Preferably, 
based on the experience of the inventor, the symbols 
should occur in different parts of a word, such as begin- 
so ning, middle, and end, throughout the pre-arranged 
script. The functionality of data processor 50 during the 
teaching process is more fully described below with ref- 
erence to Fig. 4A. 

[0042] During recognition, data processor 50 controls 
35 the recognition process which comprises choosing the 
correct symbol code corresponding to the symbol that 
was written by the user; preferably, the recognition proc- 
ess also comprises choosing the correct word which 
was written by the user. The functionality of data proc- 
40 essor 50 during the recognition process is more fully de- 
scribed below with reference to Fig. 4B. 
[0043] Reference is now additionally made to Fig. 1 B 
which is a simplified semi-pictorial semi-block diagram 
illustration of handwriting recognition apparatus con- 
45 structed and operative in accordance with an alternative 
aspect of the present invention. The apparatus of Fig. 
1B is substantially similar to the apparatus of Fig. 1A, 
except as described below. 

[0044] In the apparatus of Fig. 1 B, pen 1 0 writes on a 
so writing surface 75, which may be a sheet of paper, a 
computer display screen, or any other suitable writing 
surface. 

[0045] In the case where writing surface 75 comprises 
a computer display screen, pen 10 may optionally be 
55 employed to write without ink, so that pen 1 0 functions 
strictly as a data-input device for the associated compu- 
ter. It is appreciated that pen 10 in this case is particu- 
larly suited to function generally as a data input and 
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pointing device as well as functioning as described be* 
low for handwriting analysis. 

[0046] Pen 10 of Fig. 1B is similar to pen 10 of Fig. 
1 A and additionally comprises an ultrasonic transmitter 
80, such as model MA40S3S commercially available 5 
from Murata Electronics, GMBH, Halbeinstrabe 21-23, 
D-8500, Numberg, Germany. Ultrasonic transmitter 80 
may be mounted on the pen at a suitable location, such 
as near the tip. Alternatively, ultrasonic transmitter 80 
may be retrofittable to the pen. Further alternatively, ul- 10 
trasonic transmitter 80 may comprise a plurality of trans- 
mitters. 

[0047] A plurality of ultrasonic receivers 85, such as 
model MA40S3R commercially available from Murata 
Electronics, GMBH, Halbeinstrabe 21-23, D-8500, is 
Nurnberg, Germany, are located at predetermined po- 
sitions in proximity to writing surface 75. In the case 
where writing surface 75 comprises a typical sheet of 
paper or a typical computer display screen, two ultra- 
sonic receivers 85 are typically sufficient to determine 20 
the position of pen 10. In the case where writing surface 
75 is large compared to a typical sheet of paper or a 
typical computer display screen, more than two receiv- 
ers are preferred. 

[0048] In the case where two receivers 85 are em- 25 
ployed, the line determined by the centers of the two 
receivers is preferably substantially parallel to the plane 
in which writing surface 75 largely lies. In the case where 
more than two receivers 85 are employed, the line de- 
termined by the centers of the receivers is preferably 30 
substantially located in a plane parallel to the plane in 
which writing surface 75 largely lies. 
[0049] The receivers 85 are preferably located rela- 
tive to the writing surface 75 in such a way as to maxi- 
mize the accuracy of measurement of the position of the 35 
pen. In the case of the MA40S3R receivers referred to 
above, the preferable minimum distance between each 
of the receivers 85 and the nearest portion of writing sur- 
face 75 is twenty centimeters, and the preferable max- 
imum distance between each of the receivers 85 and *o 
the furthest portion of writing surface 75 is one meter. 
[0050] The receivers 85 typically have a preferred di- 
rection of reception; receivers 85 should preferably be 
directed in the general direction of the area in which pen 
1 0 will be employed. Also, transmitter 80 typically has a 45 
preferred direction of transmission; pen 10 should pref- 
erably be employed so that the preferred direction of 
transmission of transmitter 80 will be toward receivers 
85. 

[0051] A control circuit 90 periodically triggers ultra- so 
sonic transmitter 80, accumulates data from ultrasonic 
receivers 85, and calculates position of pen 10 based 
on the accumulated data. Control circuit 90 comprises 
a digital signal processor, known herein as a "DSP", 
such as model ADSP 21 01 commercially available from 55 
Analog Devices, One Technology Way, P.O.B. 9106, 
Norwood, MA 02062-9106, USA. 

[0052] Control circuit 90 also comprises a plurality of 



digital counters operatively associated with the plurality 
of ultrasonic receivers 85. A suitable counter is model 
74HCT4040 commercially available from RCA GMBH, 
Ridierstrasse 35A, 8000 Munchen 2, Germany. Control 
circuit 90 further comprises appropriate memory units 
for data storage and program storage. 
[0053] Control circuit 90 also amplifies and digitizes 
the acceleration data. Control circuit 90 amplifies the ac- 
celeration signals from pen 1 0 and converts them to dig- 
ital form, preferably using an analog to digital converter. 
Fig. 2B is a schematic drawing of a preferred embodi- 
ment of an analog to digital converter suitable for use in 
this application. 

[0054] The functionality of the apparatus of Fig. 1 B 
will now be described. The user employs pen 1 0 to write 
symbols on writing surface 75. Pen 10 sends accelera- 
tion data describing the accelerations of pen 1 0 during 
writing to control circuit 90. 

[0055] Control circuit 90 sends signals to pen 1 0, trig- 
gering ultrasonic transmitter 80 to emit a sequence of 
pulses. Typically, control circuit 90 sends a burst of puls- 
es, each burst separated from the next burst by a spe- 
cific time period. Preferably, four closely spaced pulses 
are sent in each burst, and the bursts are separated by " - 
25 microseconds. Reference is now additionally made 
to Fig. 2C, which is a schematic drawing of a preferred 
structure of the portion of control circuit 90 which sends 
signals to pen 10. 

[0056] Control circuit 90 restarts the counters includ- 
ed therein at the beginning of each burst. Each of the 
ultrasonic receivers 85 receives the ultrasonic sound 
waves emitted by ultrasonic transmitter 80 as part of the 
burst, and sends a signal to control circuit 90. Upon re- 
ceipt of a signal from one of the ultrasonic receivers 85, 
control circuit 90 stops the associated counter. Refer- ' 
ence is now additionally made to Figs. 2D through 2H, 
which are schematic drawings of a preferred structure 
of the portion of control circuit 90 comprising the 
counters, DSP, and associated electronic circuits. 
[0057] Control circuit 90, in parallel with the other op- 
erations carried out by control circuit 90, calculates the 
coordinates of the tip of pen 10 based on the values in 
the counters within the control circuit. Control circuit 90 
filters the counter's signals by removing large jumps in 
the counter value between one burst and the next; typ- 
ically, jumps exceeding 25 to 30 units on the counter are 
considered large. 

[0058] When a large jump comprising a single data 
point is detected by control circuit 90, the value of the 
point is recomputed as the average of the previous and 
next values. When a large jump comprises two data 
points, control circuit 90 recomputes the values using 
linear interpolation. When a large jump comprises more 
than two data points, control circuit 90 recomputes the 
values of the points after the jump by subtracting the 
height of the jump, measured over the preceding data 
points. 

[0059] Control circuit 90 removes quantization noise 
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of the counters. When a jump of one unit on the counter 
comprising a single data point is detected, control circuit 
90 recomputes the value of that data point using a zero 
order interpolation, giving the data point a value equal 
to that of the preceding and following data points. 
[0060] Control circuit 90 then computes the position 
of the tip of pen 1 0 using triangulation as follows. The 
counter value represents the time between the trans- 
mission of a pulse by ultrasonic transmitter 80 and the 
receipt of the pulse by the one of ultrasonic receivers 85 
associated with the counter. Because the speed of 
sound in air is taken to be isotropic, the counter value 
also represents the distance between ultrasonic trans- 
mitter 80 and the one of ultrasonic receivers 85 associ- 
ated with the counter. The position of pen 10 is then 
computed by applying the Pythagorean theorem. 
[0061] Control circuit 90 then filters the signals using 
a low-pass filter, such as that described in Digital Filter 
Design by T.W. Parks and C.S. Burrus, published by 
John Wiley & Sons, 1 987, chapter 7, section 7.3.3, using 
the fourth order low pass digital filter with cut-off fre- 
quency of 0.03. 

[0062] The position data thus computed by control cir- 
cuit 90 may comprise the position data used subse- 
quently by the apparatus of Fig. 1 B. Alternatively, as de- 
scribed below, position data may be computed from a 
combination of data received by ultrasonic receivers 85 
and acceleration data. 

[0063] Control circuit 90 amplifies and digitizes the ac- 
celeration data. If the acceleration data and the position 
data are not synchronized in time, control circuit 90 syn- 
chronizes the acceleration and position data by match- 
ing pen status signals with acceleration data. Pen status 
signals are described more fully below with reference to 
Fig. 4B. The combined synchronized data is sent by 
control circuit 90 to data processor 50. 
[0064] Alternatively, control circuit 90 may also com- 
prise a processor and memory suitable for storing the 
combined synchronized data. In this case, pen 10 and 
control circuit 90 may be used to produce and store the 
combined synchronized data. Then, at a later time, the 
data is sent by control circuit 90 to data processor 50 in 
order for handwriting recognition to occur. 
[0065] In the case described above in which data from 
ultrasonic receivers 85 is combined with acceleration 
data to obtain position data, the improved position data 
is computed by data processor 50. Alternatively, the new 
position data may be computed by the DSP described 
above with reference to control circuit 90. 
[0066] Data processor 50 filters the position data from 
control circuit 90 to remove components other than 
noise. For example, the acceleration data may be fil- 
tered by a Butterworth digital filter described in Digital 
Filter Design by T.W. Parks and C.S Burrus, published 
by John Wiley & Sons, 1 987, chapter 7, section 7.3.3, 
using the 4th order lowpass digital filter with cut-off fre- 
quency of 0.7 to 0.9. 

[0067] The noise component is generated when pen 



1 0, being moved by the user, is moved over irregularities 
in writing surface 75, including microscopic irregulari- 
ties. In the case of most writing surfaces 75, such as for 
example a sheet of paper, the microscopic surface ir- 

5 regularities occur at regular intervals over surface 75. 
The number of peaks in the noise component is taken 
to represent the number of surface irregularities tra- 
versed, and hence the distance traversed. 
[0068] Data processor 50 identifies and counts peaks 

10 jn the noise. The number of peaks per unit of distance 
may be determined in advance for the particular type of 
writing surface 75, may be determined by a learning 
process, or may be determined by other means. 
[0069] The positions determined by control circuit 90 

15 from ultrasound data are taken to specify the direction 
of movement, and the number of peaks in the noise 
component as described above are taken to specify the 
distance traversed. Data processor 50 computes, from 
the direction of movement and the distance traversed, 

20 updated and more accurate position data. Preferably, 
the sample rate of the acceleration data used to com- 
pute the noise is greater than the data transmission rate 
of ultrasound data from control circuit 90. 
[0070] Before handwriting recognition may occur, da- 

25 ta processor 50 must create a database which compris- 
es data about each symbol which may be recognized, 
specific to each person who uses the apparatus. The 
database is termed herein the per-person per-symbol 
database. 

30 [0071] During teaching, data processor 50 controls 
the creation of the per-person per-symbol database of 
handwriting characteristics. The per-person per-symbol 
database comprises data for each person whose hand- 
writing is known to the system, describing one or more 

35 prototypes for each symbol. Each prototype comprises 
parameters which describes the symbol as written by 
the person; the parameters are described in more detail 
below with reference to step 240. 
[0072] During teaching the user writes symbols from 

40 a pre-arranged script known to data processor 50, and 
the associated symbols appear on symbol display 60 
during the writing. Preferably, the pre-arranged script 
contains several repetitions of each symbol. Preferably, 
based on the experience of the inventor, the symbols 

45 should occur in different parts of a word, such as begin- 
ning, middle, and end, throughout the pre-arranged 
script. The functionality of data processor 50 during the 
teaching process is more fully described below with ref- 
erence to Fig. 4A. 

so [0073] During recognition, data processor 50 controls 
the recognition process which comprises choosing the 
correct symbol code corresponding to the symbol that 
was written by the user; preferably, the recognition proc- 
ess also comprises choosing the correct word which 

55 was written by the user. The functionality of data proc- 
essor 50 during the recognition process is more fully de- 
scribed below with reference to Fig. 4B. 
[0074] Reference is now additionally made to Fig. 3 
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which is a simplified block diagram of the handwriting 
recognition apparatus of Figs. 1 A and 1 B. The handwrit- 
ing recognition apparatus of Fig. 3 receives input from 
pen 10 and either graphic tablet 12 or control circuit 90. 
The data from either graphic tablet 1 2 or control circuit s 
90, representing positions of pen 1 0, is referred to herein 
as "position data". 

[0075] Pen 1 0, when moved by the user of the hand- 
writing recognition apparatus, transmits data describing 
the accelerations of pen 10 over time to teaching control w 
130 and/or handwriting recognition control 150. Pen 10 
also provides, in conjunction with graphic tablet 1 2, data 
describing the positions traced out over time by the tip 
of pen 1 0 as pen 1 0 is moved by the user over the sur- 
face of graphic tablet 1 2. Alternatively, in the case of the is 
apparatus of Fig. 1 B, position data is provided by control 
circuit 90 as described above. 

[0076] The data from pen 10 and the position data 
may be transmitted to a teaching control 1 30. Transmis- 
sion to teaching control 1 30 occurs for each person who 20 
is to use the system for handwriting recognition before 
the person uses the system for handwriting recognition 
for the first time. Transmission to teaching control 130 
also occurs when recognition errors are detected; use 
of teaching control 130 when recognition errors are de- 25 
tected is termed herein adaptive teaching. 
[0077] Teaching control 1 30 operates on the data re- 
ceived, which data represents hand movements by the 
user when writing a symbol, together with manually-pro- 
vided identification of the symbol codes that are asso- 30 
ciated with the data. Teaching control 1 30 then updates 
database 140, the per-person per-symbol database. 
Database 140 comprises prototypes of accelerations 
and symbol shape for each symbol, comprising data 
specific to each person for each symbol. 35 
[0078] Alternatively, the data from pen 1 0 and the po- 
sition data may be transmitted to handwriting recogni- 
tion control 1 50. Handwriting recognition control 1 50 op- 
erates on the data received from pen 1 0 and the position 
data to recognize the symbol represented by the move- *o 
ment of pen 10 on graphic tablet 12 or writing surface 
75. The output of handwriting recognition control 150 
comprises a list of symbol codes and their respective 
probabilities. 

[0079] A handwriting recognition post-processing cir- 45 
cuit 1 60, chooses the correct symbol code based on the 
list of symbol codes and probabilities, and on post- 
processing information which preferably comprises a 
database of previous confusions and a dictionary. The 
output of handwriting recognition post-processing circuit so 
1 60 is a symbol code or a word. 

[0080] Reference is now additionally made to Figs. 4A 
and 4B which are simplified flow charts illustrating op- 
eration of the handwriting recognition apparatus of Figs. 
1 A and 1 B. Fig. 4A illustrates the teaching process and 55 
Fig. 4B illustrates the recognition process. 
[0081 ] The preferred method of operation of the meth- 
od of Fig. 4A includes the following steps: 



[0082] STEP 210: Read position data and accelerom- 
eterdata in parallel. Preferably, the accelerometer data 
is sampled more often than the position data; for exam- 
ple, eight data points may be collected from the accel- 
erometer for each data point in the position data. 
[0083] The combined data preferably comprises the 
X,Y coordinates of the position of the tip of the pen 1 0 
in the grid of the tablet and the tablet contact status of 
pen 10, i.e. whether or not it is in contact with the tablet 
surface. The status may be operationally indicated as 
"pen up", "pen down", or "pen out of proximity with the 
tablet", in the case of the Wacom Tablet Processor spec- 
if ied above. The combined data also preferably compris- 
es the average value of the accelerometer data points 
corresponding to a single graphic tablet data point. The 
status of pen 1 0 may be derived from signals transmitted 
by graphic tablet 12, by a switch operatively associated 
with the tip of pen 1 0, as described below with reference 
to step 212, or by other means. 

[0084] STEP 212: Identify pen-surface contact termi- 
nation. The data from step 21 0 may not include the sur- 
face contact status of pen 10. In this case, the surface 
contact status of pen 1 0 may be derived from the accel- 
eration data. The acceleration data is filtered to remove' 
components other than noise. For example, the accel- • 
eration data is filtered by a Butterworth digital filter de- 
scribed in Digital Filter Design by T.W. Parks and C.S 
Burrus, published by John Wiley & Sons, 1987, chapter 
7, section 7.3.3, using the 4th order lowpass digital filter 
with cut-off frequency of 0.7 to 0.9. 
[0085] The filtered acceleration data is then integrat- 
ed over time. The slope of the integrated filtered accel- 
eration data is then analyzed to determine the point at . 
which the slope exceeds a threshold value. The point at 
which the slope exceeds the threshold value is taken to 
be the first point with status "pen down". The point at" 
which the slope falls below a threshold value is taken to 
be the first point with status "pen up"; the threshold value 
may or may not be the same as the previously described 
threshold value. In the case of determining pen status 
based on accelerometer data there is no status "pen out 
of proximity". 

[0086] The threshold values described above may be 
determined in advance for the particular type of pen and 
writing surface, may be determined by a learning proc- 
ess for the particular person, or may be determined by 
other means. 

[0087] STEP 215: Identify individual symbols and 
words. The data from the previous step is divided into 
data representing individual symbols. The status which 
comprises the statuses of "pen up" or "pen out of prox- 
imity" is termed herein "pen not down". Preferably, the 
number of consecutive data points with status of "pen 
not down", which data points represent a particular du- 
ration of the status "pen not down", is taken to indicate 
the end of a symbol or of a word. 
[0088] Typically, the duration of status "pen not down" 
within a range from 200 milliseconds to 400 milliseconds 
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is taken to indicate the end of a symbol. Duration of the 
status "pen not down" in the range from 800 millisec- 
onds to 1200 milliseconds is typically taken to indicate 
the end of a word. 

[0089] Alternatively, the end of a symbol or of a word 
may be indicated by data points which represent pen 
movements that are not part of a symbol, or by other 
means. Output data from step 215 comprises symbol 
end and word end data. 

[0090] STEP 220: Normalize combined data. The 
combined position data and the accelerometer data are 
normalized in time or by other means. 
[0091] STEP 230: Filter combined data. The normal- 
ized combined data received from the previous step is 
filtered in order to remove noise. The filtering may be 
accomplished by iterative smoothing of adjacent points 
until the total change in the signal due to a smoothing 
operation is less than the desired accuracy of the data, 
or by other suitable means. 

[0092] STEP 240: Parameterize combined data. The 
data is parameterized according to criteria which are 
chosen to represent each symbol. Reference is now ad- 
ditionally made to Figs. 5A through 5D which graphically 
illustrate some of the parameters. 
[0093] If the pen comprises a single accelerometer, 
the parameters preferably comprise the following: 

number of points before normalization; 
normalized signal of pen status; 
normalized signal of accelerations; 
sine of the angle a which angle is defined as the 
angle between the vector associated with the cur- 
rent data point (Xj,Y h AccZj) and the XY plane as 
shown in Fig. 5A; 
cosine of the angle a; 

sine of the angle p which angle is defined as the 
angle between the vector that connects the point 
before the previous point (Xj_ 2l Yj_2,AccZj_ 2 ) and the 
current point (Xj.Yj.AccZj), and the vector that con- 
nects the current point with the point after the sub- 
sequent point (X i+2 ,Y i+ 2,AccZj +2 ) in space (X.Y.Ac- 
cZ) as shown in Fig. 5B; 
and cosine of the angle p. 

[0094] Alternatively, the pen may comprise, for exam- 
ple, three accelerometers, which are not necessarily 
mutually orthogonal. If the accelerometers are not mu- 
tually orthogonal, the acceleration data may be convert- 
ed into equivalent data in a mutually orthogonal coordi- 
nate system as follows: 

[0095] Let the non-orthogonal signals be denoted by 
the vector u=(u 1 ,u 2 ,u 3 ) T and the orthogonal signals be 
denoted by the vector u'-(u\ .u'g.u^) 1 ", where T denotes 
the transpose. Then u'=AoA- 1 u where A is a vector of 
static sensitivity vectors A-(A 1 ,A 2 ,A 3 ) of the three- ac- 
celerometers. The static sensitivity vector is computed 
from the outputs of the accelerometers during a defined 
orientation without movement. Ao is a diagonalized ma- 



trix of sensitivity of the orthogonal coordinate system 
comprising the norms of A v A 2 , and A 3 . 
[0096] When the pen comprises, for example, three 
accelerometers, the parameters preferably comprise 
5 those mentioned above plus the following parameters: 

sine of the angle a' which angle is defined as the 
angle between the vector associated with the cur- 
rent data point (AccXj.AccYj.AccZj) and the AccX- 
10 AccY plane as shown in Fig. 5C; 
cosine of the angle a'; 

sine of the angle P' which angle is defined as the 
angle between the vector that connects the point 
before the previous data point (AccXj_ 2 ,AccY;_ 2 , Ac- 

15 c Zi_ 2 ) and the current point (AccXj,AccYj,AccZj), 
and the vector that connects the current point with 
the point after the subsequent point (AccX j+2 ,Ac- 
cY j+2 ,AccZ i+2 ) in space (AccX,AccY,AccZ) as 
shown in Fig. 5D; 

20 and cosine of the angle p\ 

[0097] STEP 250: Generalize parameters. The pa- 
rameters of the symbol being learned represent a spe- 
cific instance of the symbol. The symbol prototype 

25 stored by the system is to represent the general char- 
acteristics of the symbol as drawn by that person. There- 
fore, the parameters of the symbol being learned are 
generalized by some suitable means, such as by com- 
putation of the average of the value of each parameter 

30 from previous instances of the symbol along with the val- 
ue of each parameter from the current instance of the 
symbol. 

[0098] STEP 260: Update per-person per-symbol 
prototype database. The newly computed parameters 
35 from the previous step are stored in the per-person per- 
symbol prototype database. 

[0099] The preferred method of Fig. 4B includes steps 
which have been described as part of the description of 
Fig. 4 A. Description of those steps which are identical 
^o to steps in Fig. 4A have been omitted for the sake of 
brevity. The remainder of the steps in Fig. 4B include 
the following: 

[0100] STEP 300: For each prototype in the per-per- 
son per-symbol prototype database, build an index of 

45 comparison between the sample and the prototype, 
combined over parameters in the prototype. In accord- 
ance with one preferred embodiment of the present in- 
vention, all parameters are combined together to pro- 
duce the index of comparison. 

so [0101] In accordance with another device, parame- 
ters which come from graphic tablet data are combined 
and compared together, and parameters from acceler- 
ometer data are separately combined and compared, 
thus producing two measures of comparison between 

55 the sample and the prototype. Appendix B is a computer 
listing in the C programing language comprising rou- 
tines that are a preferred implementation of step 300. 
The routines comprise the following, which are found in 
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section IV, "recognition procedures": def_k_class; 
def_k_!etter; def_k_word; def_k_row; mut_metric; 
metric_dir_y; metric_dir_x; together with various defini- 
tions used by the routines. 

[0102] STEP 310: Create a list or lists of probable s 
symbols sorted by likelihood. Based on the index or in- 
dices of comparison generated in step 300, a single list 
of probable symbols sorted by likelihood is generated. 
[0103] According to a preferred embodiment of the 
present invention, two separate lists of probable sym- w 
bols sorted by likelihood are generated. When two sep- 
arate lists are generated, one comprises likely symbols 
based on graphic tablet data and the other comprises 
likely symbols based on accelerometer data. 
[0104] STEP 320: Choose the correct symbols and is 
the correct word based on the list or lists, the database 
of previous confusions and a dictionary. If one list was 
generated previously, the symbols with greatest likeli- 
hood are the candidates from which the correct symbol 
is chosen. 20 
[0105] If more than one list was generated, the lists 
are combined and symbols with high likelihood on the 
combined list are the candidates from which the correct 
symbol is chosen. Preferably, the lists may be combined 
by a voting procedure which first forms the intersection 25 
of the two lists. 

[0106] If the intersection of the two lists is empty, so 
that no symbol appears in both lists, the output is the list 
which was produced from the position data. If the inter- 
section is not empty, so that at least one symbol appears 30 
in both lists, the output list comprises all symbols includ- 
ed in the intersection group of the two input lists, sorted 
according to the average of the probabilities contained 
in the two lists. 

[0107] The database of previous confusions provides 35 
information that allows the correction of the choice of 
the correct symbol based on previous incorrect identifi- 
cations. The database of previous confusions compris- 
es, for each symbol, a list of other symbols which have 
been confused with the first symbol; for example, that *o 
the symbol "f" has often been confused with the symbol 
"b". When such an entry is found comprising previous 
confusions for a symbol in the list, the symbol or symbols 
that have previously been confused with the symbol in 
the list are added to the list. In accordance with the pre- *s 
vious example, if the symbol T is found in the list, then 
the symbol °b" is added to the list. 
[01 08] An indication of the end of each word has been 
passed as output since step 215, described above. 
Based on the indication, the most likely word, compris- so 
ing the most likely identifications for each symbol in the 
list, is identified. 

[0109] The most likely word is checked against the 
dictionary. Preferably, the dictionary comprises both a 
general dictionary used for all users of the system and 55 
a personal dictionary for each user of the system. If an 
entry exists in the dictionary for the most likely word, the 
word is chosen as the correct identification. 



[0110] If the most likely word is not found in the dic- 
tionary, all possible word combinations in the list are 
formed and each is checked against the dictionary. 
Among all such words which are found in the dictionary, 
the word with the highest likelihood is then chosen as 
the correct identification. 

[01 1 1] If none of the words is found in the dictionary, 
the most likely word is chosen as the correct identifica- 
tion. 

[0112] STEP 330: Check to see if a correction has 
been entered. During the process of recognition, the us- 
er of the system is preferably provided with a visual in- 
dication of each symbol recognized. 
[01 1 3] After the end of a word is detected, the user of 
the system preferably is provided with a visual indication 
of the word recognized. The user may indicate manually 
that a given word was incorrectly recognized and may 
input a correction. 

[0114] STEP 340: Update database of previous con- 
fusions. Based on a manual correction entered in step 
330 or an automatic correction based on the dictionary, 
the database of previous confusions is updated. Based 
on a manual correction, the personal dictionary is also- 
updated if the corrected word is not found in the diction- 
ary. 

[01 1 5] Reference is now additionally made to Fig. 6A- 
which is a simplified illustration of a preferred mechan- 
ical structure of the digitizing pen/accelerometer 10 of 
Fig. 1A. Pen 10 may comprise an electromagnetic pen 
such as a WACOM SP-200 available from WACOM 
Computer Systems GmbH, Hellersbergstrasse 4, W- ' 
4040 Neuss 1 , Germany, or any other suitable pen de- 
signed for use with a suitable model of graphic tablet:- 
Pen 10 also comprises an electromagnetic coil 410" 
which operates in conjunction with graphic tablet 12 to*- 
provide and indication of the location of the point of pen " 
1 0. Pen 1 0 further comprises an ink refill 420 which pro- 
vides the ability to write on paper or on another surface. 
[0116] Pen 10 also comprises three built-in mutually 
orthogonal accelerometers 25 such as ICS 3031-2 
available from IC Sensors. Alternatively, there may be 
a plurality of accelerometers not necessarily orthogonal- 
ly disposed. According to a still further alternative, there 
may be only one accelerometer. 

[0117] Pen 10 also comprises associated amplifiers 
30, described above with reference to Fig. 1 A. Pen 10 
further comprises communications cable 450 which is 
used to send data to control circuit 40. Alternatively, the 
data may be sent through any kind of appropriate com- 
munication link, such as ultrasonic, infrared, or by any 
other suitable means. 

[01 1 8] Reference is now additionally made to Fig. 6B 
which is a simplified illustration of a preferred mechan- 
ical structure of the digitizing pen/accelerometer 10 of 
Fig. 1B. Pen 10 may comprise a housing such as an 
ABS plastic housing or other suitable housing. Pen 10 
further comprises an ink refill 420 and associated spring 
which provides the ability to write on paper or on another 
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surface. 

[0119] Pen 10 also comprises three built-in mutually 
orthogonal accelerometers 25 such as ICS 3031-2 
available from IC Sensors. Alternatively, there may be 
a plurality of accelerometers not necessarily orthogonal- 5 
ly disposed. According to a still further alternative, there 
may be only one accelerometer. 
[0120] Pen 10 also comprises associated amplifiers 
30, described above with reference to Fig. 1 A. 
[0121] Pen 10 also comprises ultrasonic transmitter 
80 as specified above. Pen 1 0 further comprises com- 
munications cable 450 which is used to send data to 
control circuit 90. Alternatively, the data may be sent 
through any kind of appropriate communication link, 
such as ultrasonic, infrared, or by any other suitable 
means. 

[01 22] Reference is now made to Fig. 7 which is a sim- 
plified pictorial illustration of another alternative pre- 
ferred embodiment of handwriting recognition appara- 
tus constructed and operated in accordance with the 
present invention. Pen 1 0 is employed to write in a note- 
book, on a sheet of paper, or on any other suitable sur- 
face. In the apparatus of Fig. 7, Pen 1 0 is as described 
above but the one or more accelerometers 25 comprise 
a plurality of accelerometers. 

[0123] The apparatus of Fig. 7 includes a scanner 
500, such as a Logitech ScanMan, model 256. The user 
writes with pen 10 on writing surface 512. The descrip- 
tion above in reference to Fig. 1 also applies to Fig. 7, 
except as described herein with reference to Fig. 7. 
[0124] Pen 10 sends acceleration data through con- 
trol circuit 40 to data processor 50. After writing is con- 
cluded, surface 512 is scanned using scanner 500. 
Teaching and recognition then occur based on the com- 
bined data from pen 10 and scanner 500. 
[0125] Reference is now additionally made to Fig. 8 
which is a simplified block diagram of the handwriting 
recognition apparatus of Fig. 7. The handwriting recog- 
nition apparatus of Fig. 8 receives input from pen 1 0 and 
scanner 500. 

[0126] Pen 1 0, when moved by the user of the hand- 
writing recognition apparatus, transmits data describing 
the accelerations of pen 1 0 over time to acceleration 
teaching control 630 and/or acceleration handwriting 
recognition control 650. 

[01 27] The term OCR as used herein refers to optical 
character recognition. Scanner 500, when it scans writ- 
ing surface 51 2, transmit data describing the image on 
surface 512 to OCR teaching control 635 and/or OCR 
control 655. 

[0128] The data from pen 10 and scanner 500 may be 
transmitted to acceleration teaching control 630 or OCR 
teaching control 635 respectively. Transmission to ac- 
celeration teaching control 630 or OCR teaching control 
635 occurs for each person who is to use the system for 
handwriting recognition before the person uses the sys- 
tem for handwriting recognition for the first time. Trans- 
mission to acceleration teaching control 630 or OCR 



teaching control 635 also occurs when recognition er- 
rors are detected; use of acceleration teaching control 
630 or OCR teaching control 635 when recognition er- 
rors are detected is termed herein adaptive teaching. 
[0129] Acceleration teaching control 630 or OCR 
teaching control 635 operates on the data received, 
which data represents hand movements by the user 
when writing a symbol, together with manually-provided 
identification of the symbol codes that are associated 
with the data. Acceleration teaching control 630 then up- 
dates database 640, a per-person per-symbol acceler- 
ation database; OCR teaching control 635 updates da- 
tabase 645, a per-person per-symbol OCR database. 
Database 640 and database 645 comprise prototypes 
of accelerations and symbol shape for each symbol, 
comprising data specific to each person for each sym- 
bol. 

[0130] Alternatively, the data from pen 10 and scan- 
ner 500 may be transmitted to acceleration handwriting 
recognition control 650 or OCR control 655 respectively. 
Acceleration handwriting recognition control 650 and 
OCR control 655 operate on the data received from pen 
10 and scanner 500 respectively to recognize the sym- 
bol represented by the movement of pen 1 0 on scanner 
500. 

[0131] The output of acceleration handwriting recog- 
nition control 650 comprises a list of symbol codes and 
their respective probabilities. An acceleration handwrit- 
ing recognition post-processing circuit 660, chooses the 
correct symbol code based on the list of symbol codes 
and probabilities, and on post-processing information 
which preferably comprises a database of previous con- 
fusions and a dictionary. The output of acceleration 
handwriting recognition post-processing circuit 660 is a 
list of symbol codes and/or words sorted by likelihood. 
[0132] Recognized word storage circuit 670 stores, 
for each word recognized, the sorted list of likely words 
output by acceleration handwriting recognition post- 
processing circuit 660. The stored lists are kept for later 
processing. 

[0133] The output of OCR control 655 comprises, for 
each symbol in the scanned image from scanner 500, a 
list of probable symbol codes sorted by likelihood. OCR 
control 655 may be implemented, for example, by any 
appropriate commercially available OCR system. 
[0134] Word/string matcher 680 accepts as input the 
lists of words output by acceleration handwriting recog- 
nition post-processing circuit 660 and the list of probably 
symbol codes output by OCR control 655. Word/string 
matcher 680 finds matches between its two input lists. 
The output of word/string matcher 680 comprises a list 
of words and/or strings of symbols and the position of 
each word and/or string on writing surface 512. 
[0135] Reference is now additionally made to Figs. 9A 
and 9B which are simplified flow charts illustrating op- 
eration of the handwriting recognition apparatus of Fig. 
7 in accordance with a preferred embodiment of the in- 
vention. Fig. 9A illustrates the teaching process and Fig. 
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9B illustrates the recognition process. 
[01 36] The preferred method of Fig. 9A includes steps 
which have been described as part of the description of 
Fig. 4A. Description of those steps which are identical 
to steps in Fig. 4A have been omitted for the sake of 5 
brevity. The remainder of the steps in Fig. 9A include 
the following: 

[0137] STEP 71 0: Read accelerometer data. The ac- 
celerometer data comprises data points representing 
sampling of the acceleration measured by accelerome- 10 
ters 25. Preferably, the sampling rate is approximately 
1600 data points per second, averaged over 8 points, 
producing an output of approximately 200 data points 
per second. 

[0138] STEP 712: Identify pen-surface contact termi- 15 
nation. The data from step 71 0 does not include the sur- 
face contact status of pen 1 0. The surface contact status 
of pen 1 0 may be derived from the acceleration data, as 
explained in step 212 above with reference to tablet con- 
tact data. 20 
[0139] STEP 715: Identify individual symbols and 
words. The data from the previous step is divided into 
data representing individual symbols. The status which 
comprises the status of "pen up" is termed herein "pen 
not down". Preferably, the number of consecutive data 25 
points with status of "pen not down", which data points 
represent a particular duration of the status "pen not 
down", is taken to indicate the end of a symbol or of a 
word. 

[01 40] Typically, the duration of status "pen not down" 30 
within a range from 200 milliseconds to 400 milliseconds 
is taken to indicate the end of a symbol. Duration of the 
status "pen not down" in the range from 800 millisec- 
onds to 1200 milliseconds is typically taken to indicate 
the end of a word. 35 
[0141] Alternatively, the end of a symbol or of a word 
may be indicated by data points which represent pen 
movements that are not part of a symbol, or by other 
means. Output data from step 715 comprises symbol 
end and word end data. *o 
[0142] STEP 720: Normalize accelerometer data. 
The accelerometer data is normalized in time or by other 
means. 

[0143] STEP 730: Filter accelerometer data. The nor- 
malized accelerometer data received from the previous 45 
step is filtered in order to remove noise. The filtering may 
be accomplished by iterative smoothing of adjacent 
points until the total change in the signal due to a 
smoothing operation is less than the desired accuracy 
of the data, or by other suitable means. so 
[0144] STEP 740: Parameterize accelerometer data. 
The data is parameterized according to criteria which 
are chosen to represent each symbol. If the accelerom- 
eters are not mutually orthogonal, the acceleration data 
may be converted into equivalent data in a mutually or- 55 
thogonal coordinate system as follows as described 
above with reference to step 240. 
[0145] The parameters preferably comprise the fol- 



lowing: 

number of points before normalization; 

normalized signal of pen status; 

normalized signal of Z acceleration; 

sine of the angle a' defined above with reference to 

step 240; 

cosine of the angle a'; 

sine of the angle P' defined above with reference to 
step 240; 

and cosine of the angle p\ 

[0146] STEP 760: Update per-person per-symbol ac- 
celeration prototype database. The newly computed pa- 
rameters from the previous step are stored in the per- 
person per-symbol acceleration prototype database. 
[0147] STEP 770: Has the end of the predefined text 
been reached? Whether the end has been reached is 
determined by whether or not the end of the predefined 
text file has been reached. If not, the process continues 
with step 71 0. If the end has been reached, the process 
continues with step 780. 

[0148] STEP 780: Scan written text. The text written ~ 
on surface 512 is scanned using scanner 500. 
[0149] STEP 790: OCR teaching process. For each 
symbol in the text, features of the corresponding writing ■ 
on surface 51 2 are identified. 

[0150] STEP 795: Update per-person per-symbol 
OCR prototype database. Features identified in step 
790 are updated in the per-person per-symbol OCR pro- 
totype database. 

[01 51 ] The preferred method of Fig. 9B includes steps* . 
which have been described as part of the description of* 
Figs. 4A and 9A. Description of those steps which are ^ 
identical to steps in Figs. 4A and 9A have been omitted, * 
for the sake of brevity. The remainder of the steps in Fig. 
9B include the following: 

[0152] STEP 800: For each prototype in the per-per- 
son per-symbol acceleration prototype database, build 
an index of comparison between the sample and the 
prototype, combined over parameters in the prototype. 
In accordance with a preferred embodiment of the 
present invention, all parameters are combined together 
to produce the index of comparison. 
[0153] STEP 810: Create a list of probable symbols 
sorted by likelihood. Based on the index or indices of 
comparison generated in step 800, a single list of prob- 
able symbols sorted by likelihood is generated. 
[0154] STEP 820: Choose the correct symbols and 
the correct word based on the list, the database of pre- 
vious confusions and a dictionary. The symbols with 
greatest likelihood are the candidates from which the 
correct symbol is chosen. 

[01 55] As described above with reference to step 320, 
the database of previous confusions provides informa- 
tion that allows the correction of the choice of the correct 
symbol based on previous incorrect identifications. 
[01 56] An indication of the end of each word has been 
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passed as output since step 715, described above. 
Based on the indication, the most likely word, compris- 
ing the most likely identifications for each symbol in the 
list, is identified. 

[0157] The most likely word is checked against the 5 
dictionary. Preferably, the dictionary comprises both a 
general dictionary used for all users of the system and 
a personal dictionary for each user of the system. If an 
entry exists in the dictionary for the most likely word, the 
word is chosen as the correct identification. 
[0158] If the most likely word is not found in the dic- 
tionary, all possible word combinations in the list are 
formed and each is checked against the dictionary. 
Among all such words which are found in the dictionary, 
the word with the highest likelihood is then chosen as 
the correct identification. 

[0159] If none of the words is found in the dictionary, 
the most likely word is chosen as the correct identifica- 
tion. 

[0160] STEP 822: Store recognition information. The 
sorted list of likely words output by the previous step is 
stored for later processing. 

[0161] STEP 824: Check for end of text. The user in- 
dicates end of text using input device 70. 
[0162] STEP 826: Scan written text. Scanner 500 is 
employed to scan writing surface 51 2. The scanned im- 
age is output to step 828. 

[0163] STEP 828: Perform OCR. OCR control 655 is 
employed to perform optical character recognition on 
the scanned image from the preceding step. The output 
of step 828 comprises, for each symbol in the scanned 
image from step 826, a list of probable symbol codes 
sorted by likelihood and the position of each symbol in 
the scanned image. OCR control 655 may be imple- 
mented, for example, by any appropriate commercially 
available OCR system. 
[0164] STEP 830: Find matching words 
[01 65] For each word from acceleration, find all exact- 
ly matching words in OCR and list them including loca- 
tion; or mark no match. 

[0166] For each word in the acceleration list where 
matches were found, succeeding words are checked to 
see whether the succeeding words correspond to words 
in succeeding locations as indicated from the OCR list; 
this process continues until no further match is found. 
There may be several possible matches for each word 
in the acceleration list. Further matching sequences of 
words are sought beginning with the next word after the 
last word of the preceding list of matches. 
[01 67] For each remaining unmatched word in the ac- 
celeration list, the closest matching word in the OCR list 
is identified based on some measure of variance. For 
example, the measure of variance may be the number 
of matching letters between the two words on the list, 
with a minimum of 2 letters matching. 
[0168] The regions adjacent to the partially matched 
words are then checked; if the region before and after 
are matched, then we decide that the partially matched 



words are matched. 

[0169] The unmatched words in the OCR list are now 
examined. For any remaining unmatched words, if the 
region before and after the word matches, the word is 
considered to match. 

[0170] Optionally, any other context sensitive meth- 
ods such as grammar checking or other appropriate 
means of identifying missing sections may be used to 
identify still unmatched words. Any remaining un- 
matched words are identified manually by the user. 
[0171] STEP 840: Update database of previous con- 
fusions. Based on a manual correction entered in step 
330 or an automatic correction based on the dictionary, 
the database of previous confusions is updated. Based 
on a manual correction, the personal dictionary is also 
updated if the corrected word is not found in the diction- 
ary. 

[01 72] It is appreciated that various features of the in- 
vention which are, for clarity, described in the contexts 
of separate embodiments may also be provided in com- 
bination in a single embodiment. Conversely, various 
features of the invention which are, for brevity, described 
in the context of a single embodiment may also be pro- 
vided separately or in any suitable subcombination. 
[0173] It will be appreciated by persons skilled in the 
art that the present invention is not limited to what has 
been particularly shown and described hereinabove. 
Rather, the scope of the present invention is defined on- 
ly by the claims that follow: 



Claims 

1. Handwriting recognition apparatus comprising 

handwriting input apparatus employing at 
least two different sensing techniques to sense 
handwriting and including at least one accelerome- 
ter (25) located in said hand held writing implement 
(1 0) other than at a tip thereof; and 

symbol identification apparatus (50) receiving 
an output of the handwriting input apparatus for pro- 
viding an output indication of symbols represented 
by the handwriting, and wherein said symbol iden- 
tification apparatus includes a first identification 
channel for employing an input, different from said 
at least one accelerometer (25) from said handwrit- 
ing input apparatus to provide a first output repre- 
senting identification of symbols, a second identifi- 
cation channel for employing an input from said at 
least one accelerometer to provide a second output 
representing identification of symbols, and symbol 
selection apparatus for selecting between the first 
and the second outputs based on predetermined 
criteria. 

2. Handwriting recognition apparatus as claimed in 
claim 1 , wherein said handwriting input apparatus 
comprises a position digitizer (12). 
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3. Handwriting recognition apparatus as claimed in 
claim 2, wherein said at least one accelerometer 
(25) comprises a plurality of mutually orthogonally 
disposed accelerometers. 

5 

4. Handwriting recognition apparatus as claimed in 
claim 2, wherein said symbol identification appara- 
tus (50) includes combined position and accelera- 
tion processing apparatus, which combines the in- 
puts from said digitizer (12) and said at least one 10 
accelerator (25) in identifying symbols. 

5. Handwriting recognition apparatus as claimed in 
claim 1 , wherein said handwriting input apparatus 
comprises ultrasonic position determination appa- 15 
ratus operative in air. 

6. Handwriting recognition apparatus as claimed in 
claim 5, wherein said handwriting input apparatus 
is operative to write on a computer screen (60). 20 

7. A handwriting recognition method comprising; 

employing at least one accelerometer (25) lo- 
cated in a hand writing implement (10) other 25 
than at the tip thereof and at least one digitizer 
based on an additional different sensing tech- 
nique to sense handwriting; and 
receiving an output indication of the handwrit- 
ing and providing an output indication of sym- 30 
bols represented by the handwriting, 



wherein said step of providing an output indi- 
cation of symbols employs input from a digitizer ( 1 2) 
to provide a first output representing identification 
of symbols along a first identification channel and 
employs input from said at least one accelerometer 
(25) to provide a second output representing iden- 
tification of symbols along a second identification 
channel and selects between the first and second 
outputs based on predetermined criteria. 



claim 7, wherein said step of receiving includes, em- 
ploying digitized position inputs to provide a first 
output representing identification symbols, employ- 
ing acceleration information to provide a second 
output representing identification of symbols, and 
selecting between said first and second outputs. 



Patentanspruche 

1. Handschrifterkennungsvorrichtung, umfassend: 

eine Handschrifteingabeeinrichtung, die we* 
nigstens zwei verschiedene Erfassungstechni- 
ken zum Erfassen einer Handschrift verwendet 
und wenigstens ein Beschleunigungsmessge- 
rat (25) umfasst, das in dem handgehaltenen 
Schreibelement (10) an einer anderen Stelle 
als an einer Spitze davon angeordnet ist; und 

eine Symbolidentifikationsvorrichtung (50), die 
einen Ausgang der Handschrifteingabevorrich- 
tung zum Bereitstellen einer Ausgangsanzeige 
von Symboten, dargestellt durch die Hand- 
schrift, empfangt, und wobei die Symbolidenti- 
fikationsvorrichtung einen ersten Identifikati- 
onskanal zum Verwenden eines Eingangs, der 
sich von dem wenigstens einen Beschleuni- 
gungsmessgerat (25) unterscheidet, von der 
Handschrifteingabevorrichtung zum Bereitstel- 
len eines ersten Ausgangs, der eine Identifika- - 
tion von Symbolen darstellt, einen zweiten 
Identifikationskanal zum Verwenden eines Ein- " 
gangs von dem wenigstens einen Beschleuni- 
gungsmessgerat zum Bereitstellen eines zwei- 
ten Ausgangs, der eine Identifikation von Sym- 
bolen darstellt, und eine Symbolwahlvorrich- 
tung zum Wahten zwischen dem ersten und 
dem zweiten Ausgang auf Grundlage von vor- 
gegebenen Kriterien umfasst. 

2. Handschrifterkennungsvorrichtung nach Anspruch 
35 1, wobei die Handschrifteingabevorrichtung einen ' 

Positions-Digitalisierer (12) umfasst. 

3. Handschrifterkennungsvorrichtung nach Anspruch 
2, wobei das wenigstens eine Beschleunigungs- 

40 messgerat (25) eine Vielzahl von zueinander ortho- 
gonal angeordneten Beschleunigungsmessgera- 
ten umfasst. 

Handschrifterkennungsvorrichtung nach Anspruch 
2, wobei die Symbolidentifikationsvorrichtung (50) 
eine kombinierte Positions- und Beschleunigungs- 
verarbeitungsvorrichtung umfasst, die die Eingan- 
ge von dem Digitalisierer (12) und dem wenigstens 
einen Beschleunigungsmessgerat (25) beim Iden- 
tifizieren von Symbolen kombiniert. 

Handschrifterkennungsvorrichtung nach Anspruch 
1, wobei die Handschrifteingabevorrichtung eine 
Ultraschallpositionsbestimmungsvorrichtung, die in 
55 Luft betreibbar ist, umfasst. 

6. Handschrifterkennungsvorrichtung nach Anspruch 
5, wobei die Handschrifteingabevorrichtung be- 



8. A handwriting recognition method as claimed in 
claim 7, wherein said at least one additional sensing 4. 
technique comprises position digitizing. 45 

9. A handwriting recognition method as claimed in 
claim 8 and further comprising identifying symbols 
by combining digitized position inputs and sensed 
acceleration information. 50 

10. A handwriting recognition method as claimed in 5. 



13 



25 



EP 0 666 543 B1 



26 



treibbar ist, um auf einem Computerbildschirm (60) 
zu schreiben. 

7. Handschrifterkennungsverfahren, umfassend die 
folgenden Schritte: 5 

Verwenden wenigstens eines Beschleuni- 
gungsmessgerats (25), das in einem Hand- 
schriftelement (10) an einer anderen Stelle als 
an der Spitze davon angeordnet ist, und wenig- w 
stens eines Digital is ierers, der auf einer zusatz- 
lichen unterschiedlichen Erfassungstechnik 
basiert, zum Erfassen einer Handschrift; 

Empfangen einer Ausgangsanzeige uber die is 
Handschrift und Bereitstellen einer Ausgangs- 
anzeige von Symbolen, die von der Handschrift 
dargestellt werden, 

wobei der Schritt zum Bereitstellen einer Aus- 20 
gangsanzeige von Symbolen einen Eingang von ei- 
nem Digitalisierer (12) zum Bereitstellen eines er- 
sten Ausgangs, der eine Identifikation von Symbo- 
len darstellt, entlang eines ersten Identifikationska- 
nals verwendet und einen Eingang von dem wenig- 25 
stens einen Beschleunigungsmessgerat (25) ver- 
wendet, um einen zweiten Ausgang, der eine Iden- 
tifikation von Symbolen darstellt, entlang eines 
zweiten Identifikationskanals bereitzustellen, und 
zwischen den ersten und zweiten Ausgangen auf 30 
Grundlage von vorgegebenen Kriterien wahlt. 



un appareil d'entree d'ecriture manuelle em- 
ployant au moins deux techniques de detection 
differentes pour detecter de I'ecriture manuelle 
et incluant au moins un accelerometre (25) pla- 
ce dans {'instrument d'ecriture (10) tenu a la 
main, ailleurs qu'a une pointe de celui-ci; et 
un appareil d'identification desymboles (50) re- 
cevant une information de sortie de I'appareil 
d'entree d'ecriture manuelle pour fournir une in- 
dication de sortie de symboles representes par 
I'ecriture manuelle, et dans lequel I'appareil 
d'identification de symboles comprend un pre- 
mier canal d'identification pour employer une 
information d'entree, differente de celle prove- 
nant de I'au moins un accelerometre (25), pro- 
venant de I'appareil d'entree d'ecriture manuel- 
le, pour fournir une premiere information de 
sortie representant Identification de symboles, 
un second canal d'identification pour employer 
une information d'entree provenant de I'au 
moins un accelerometre pour fournir une se- 
conde information de sortie representant Iden- 
tification de symboles, et un appareil de selec- 
tion de symboles pour effectuer une selection 
entre les premiere et seconde informations de 
sortie, sur la base de criteres predetermines. 

2. Appareil de reconnaissance d'ecriture manuelle se- 
lon la revendication 1 , dans lequel I'appareil d'en- 
tree d'ecriture manuelle comprend un numeriseur 
de position (12). 



8. Handschrifterkennungsverfahren nach Anspruch 7, 
wobei die wenigstens eine zusatzliche Erfassungs- 
technik eine Positionsdigitalisierung umfasst. 

9. Handschrifterkennungsverfahren nach Anspruch 8 
und ferner umfassend eine Identifizierung von Sym- 
bolen durch Kombinieren von digitalisierten Positi- 
onseingangen und einer erfassten Beschleuni- 
gungsinformation. 

10. Handschrifterkennungsverfahren nach Anspruch 7, 
wobei der Schritt zum Empfangen ein Verwenden 
von digitalisierten Positionseingangen zum Bereit- 
stellen eines ersten Ausgangs, der Identifikations- 
symbole darstellt, ein Verwenden von Beschleuni- 
gungsinformation zum Bereitstellen eines zweiten 
Ausgangs, der eine Identifikation von Symbolen 
darstellt, und ein Wahlen zwischen den ersten und 
zweiten Ausgangen umfasst. 



3. Appareil de reconnaissance d'ecriture manuelle se- 
lon la revendication 2, dans lequel I'au moins un ac- 

35 celerometre (25) comprend une multiplicity d'acce- 
lerometres disposes de facon mutuellement ortho- 
gonale. 

4. Appareil de reconnaissance d'ecriture manuelle se- 
40 ion la revendication 2, dans lequel I'appareil d'iden- 
tification de symboles (50) comprend un appareil de 
traitement de position et deceleration combine, 
qui combine les informations d'entree provenant du 
numeriseur (12) et de I'au moins un accelerometre 

45 (25) dans ('identification de symboles. 

5. Appareil de reconnaissance d'ecriture manuelle se- 
lon la revendication 1 , dans lequel I'appareil d'en- 
tree d'ecriture manuelle comprend un appareil de 

50 determination de position par ultrasons, fonction- 
nant dans lair. 



40 



Revendications 

1. Appareil de reconnaissance d'ecriture manuelle 
comprenant: 



6. Appareil de reconnaissance d'ecriture manuelle se- 
lon la revendication 5, dans lequel I'appareil d'en- 

55 tree d'ecriture manuelle fonctionne de fagon a ecrire 
sur un ecran d'ordinateur (60). 

7. Un procede de reconnaissance d'ecriture manuelle 
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comprenant: 

I'emploi d'au moins un accelerometre (25) pla- 
ce dans un instrument d'ecriture manuelle (1 0), 
ailleurs qu'a la pointe de celui-ci, et au moins 5 
un numeriseur base sur une technique de de- 
tection differente supplementaire pour detecter 
I'ecriture manuelle; et 

la reception d'une indication de sortie de I'ecri- 
ture manuelle et la fourniture d'une indication u* 
de sortie de symboles representes par I'ecriture 
manuelle, 

dans lequel I'etape consistant a fournir une in- 
dication de sortie de symboles utilise une informa- is 
tion d'entree provenant d'un numeriseur (12) pour 
fournir une premiere information de sortie represen- 
tant ('identification de symboles sur un premier ca- 
nal d'identification, et utilise une information d'en- 
tree provenant de I'au moins un accelerometre (25) 20 
pour fournir une seconde information de sortie re- 
presentant I'identification de symboles sur un se- 
cond canal d'identification, et elle effectue une se- 
lection entre les premiere et seconde informations 
de sortie sur la base de criteres predetermines. 25 

8. Un procede de reconnaissance d'ecriture manuelle 
selon la revendication 7, dans lequel I'au moins une 
technique de detection supplementaire comprend 

la numerisation de position. 30 

9. Un procede de reconnaissance d'ecriture manuelle 
selon la revendication 8 et comprenant en outre 
I'identification de symboles en combinant des infor- 
mations d'entree de position numerisees et une in- 35 
formation deceleration detectee. 

10. Un procede de reconnaissance d'ecriture manuelle 
selon la revendication 7, dans lequel I'etape de re- 
ception comprend I'emploi d'tnformations d'entree *o 
de position numerisees pour fournir une premiere 
information de sortie representant I'identification de 
symboles, I'emploi d'une information d'acceleration 
pour fournir une seconde information de sortie re- 
presentant I'identification de symboles, et la selec- *s 
tion entre les premiere et seconde informations de 
sortie. 
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